-
Notifications
You must be signed in to change notification settings - Fork 2.5k
SwiftTesting: Include metadata for tags, bug, and time limit traits in event stream #3040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
ad4afed to
bf363fb
Compare
proposals/testing/NNNN-include-tags-bugs-and-timeline-in-event-stream.md
Outdated
Show resolved
Hide resolved
8a7e8b1 to
e7325c8
Compare
55d172e to
f56f077
Compare
proposals/testing/NNNN-include-tags-bugs-and-timeline-in-event-stream.md
Outdated
Show resolved
Hide resolved
| + "title": <string> ; the human readable bug title | ||
| +} ; | ||
| + | ||
| +<characters> ::= "" | <characters> <character> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to spell these out. Please remove them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK
|
|
||
| <test-id> ::= <string> ; an opaque string representing the test case | ||
| + | ||
| +<tag> ::= "." <chacters> | <tag> ; a string representation of a tag |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is invalid syntax. What are you trying to describe here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to describe that a tag has this syntax .<charaters>[.<characters|.....]
ie: .Foo.bar.bar
But this conversation might be moot based on the output of https://github.com/swiftlang/swift-evolution/pull/3040/files#r2607028858
| } | ||
| ], | ||
| "tags": [ | ||
| ".blue", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should think about whether the leading period should be encoded in the JSON. Since this schema may be used by other libraries that don't use Swift's dot syntax, my gut says they shouldn't be encoded and we should just define <tag> ::= <string>.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should think about whether the leading period should be encoded in the JSON
I don't have a strong opinion here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree w/ @grynspan and think we shouldn't necessarily specify in the schema itself that there will always be a leading period character. But I do think the proposal should clearly specify whether, in the encoded string, the leading period and any subsequent punctuation will be preserved as it was written in source. The main thing I'm trying to convey is that we ought to clearly define the behavior, whatever it is.
And on this specific point, I would suggest you define the behavior such that it will precisely preserve the way the tag was spelled in source. So if the code had .tags(.blue) the string would be ".blue". If it had .tags(Tag.red), the string would be "Tag.red", and so on. The latter example differs from the example you show in the current proposal, so that represents a change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be the wrong abstraction though, as it would mean Tag.red and .red don't compare equal. (I don't think that string is available at runtime anyway!) Furthermore, preserving the source would prevent the use of this feature with libraries that define tags using different syntax. For example, C++ might have something using ::, or Objective-C might just use @"red".
I'd like to just define it formally as <string> and document that Swift Testing specifically normalizes to "red" since that would be most compatible with what other libraries or tagging syntaxes would reasonably do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm open to that. To be comprehensive about this, how would examples which include interstitial punctuation, like .tags(.NestedType.bar, .NestedType.AnotherNestedType.baz) be encoded in the JSON? As "NestedType.bar" and "NestedType.AnotherNestedType.baz", respectively, or something else?
In effect, this will mean we'll be treating the optional (Testing.)Tag type namespace prefixes as "special", and stripping them when present, along with the leading period. Which is fine, but it should be spelled out in the proposal. (Maybe it would be helpful to include a Markdown table with several of these examples.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @grynspan that Tag.red and .red should be represented the same way in the event stream. Do we want to consider having the JSON represents each tags as an array, or maybe an object, instead of string? This might allow some IDE's to represent a tree structure of all tags?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm in favor of the proposal's current behavior for the two examples it shows—that .tags(.blue) and .tags(Tag.red) are represented as ".blue" and ".red", respectively.
Can we include one more example, just to clarify the handling of nested tags? Specifically, can we document that a two-level tag like .tags(.Foo.bar) would map to the string ".Foo.bar"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, hold on. More investigation has revealed an interesting data point: the sourcekit-lsp indexer currently does not include the leading period, so that suggests omitting it in the JSON event stream representation would make things align better and ultimately, that's one of the biggest considerations.
So here's my current preference:
| Trait | String |
|---|---|
.tags(.blue) |
"blue" |
.tags(Tag.red) |
"red" |
.tags(.Foo.bar) |
"Foo.bar" |
.tags(Tag.Foo.bar) |
"Foo.bar" |
.tags(Testing.Tag.Foo.bar) |
"Foo.bar" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy to use a string with the leading . removed. Do we know where sourcekit-lsp indexer gets the value from? Does it generate that string itself? Is there value in creating a Swift Testing API that would be the source of truth, which sourcekit-lsp can then eventually start using?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's conceivable, but pretty far outside the scope of this proposal so I'd recommend breaking that out into a separate discussion
proposals/testing/NNNN-include-tags-bugs-and-timeline-in-event-stream.md
Outdated
Show resolved
Hide resolved
proposals/testing/NNNN-include-tags-bugs-and-timeline-in-event-stream.md
Show resolved
Hide resolved
f56f077 to
0a650cd
Compare
7530592 to
113f1a1
Compare
113f1a1 to
71d2c9b
Compare
| .tags(Tag.red), | ||
| .bug("https://my.defect.com/1234"), | ||
| .bug("other defect"), | ||
| .timeLimit(Swift.Duration.seconds(testTimeLimit + 100)), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI these three aren't valid. Use .minutes() instead (the overloads that take Swift.Duration are unavailable.)
|
|
||
| ```json | ||
| { | ||
| "kind": "test", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick: missing the "version" field.
| public static var blue: Self { | ||
| Tag(kind: .staticMember("blue")) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| public static var blue: Self { | |
| Tag(kind: .staticMember("blue")) | |
| } | |
| @Tag public static var blue: Self |
| public static var red: Self { | ||
| Tag(kind: .staticMember("red")) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| public static var red: Self { | |
| Tag(kind: .staticMember("red")) | |
| } | |
| @Tag public static var red: Self |
| .tags(.blue), | ||
| .tags(Tag.red), | ||
| .bug("https://my.defect.com/1234"), | ||
| .bug("other defect"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| .bug("other defect"), | |
| .bug(id: "12345", "other defect"), |
The compiler will reject an invalid URL string here rather than interpreting it as a comment.
| "url": "https:\/\/my.defect.com\/1234" | ||
| }, | ||
| { | ||
| "url": "other defect" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| "url": "other defect" | |
| "id": "12345" | |
| "title": "other defect" |
| <...SNIP...>, | ||
| "bugs": [ | ||
| { | ||
| "url": "https:\/\/my.defect.com\/1234" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| "url": "https:\/\/my.defect.com\/1234" | |
| "url": "https://my.defect.com/1234" |
If we're escaping forward slashes right now, that's a bug on our side I think.
| - **`timeLimitInSeconds` vs `timeLimit`**: We chose the shorter `timeLimit` name for | ||
| consistency with Swift Testing's existing API, with the time unit documented in the | ||
| schema specification. The naming convention was discussed with the Testing Workgroup | ||
| and it was decided that a seperata proposal should be made on how to represent |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| and it was decided that a seperata proposal should be made on how to represent | |
| and it was decided that a seperate proposal should be made on how to represent |
|
|
||
| ## Future directions | ||
|
|
||
| This enhancement establishes future richer tooling experiences: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| This enhancement establishes future richer tooling experiences: |
I don't think this sentence is meaningful.
|
|
||
| No existing functionality is affected, making this a purely additive enhancement. | ||
|
|
||
| ## Integration with supporting tools |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd suggest reworking this section. The leading sentence doesn't say anything and then bullet points are vague and promise functionality that's beyond the scope of this proposal. Perhaps something like:
"Supporting tools like IDEs and CI systems consume the JSON event stream and will be able to use this new information to provide richer developer experiences. For example, an IDE could allow developers to sort the content of a test report by tag, or could highlight failing tests with associated bugs that are incorrectly marked fixed."
(But feel free to rewrite that of course, that's just off the top of my head.)
| - **Performance Monitoring**: Can track and alert on time limit violations | ||
| - **Bug Tracking Integration**: Can correlate test results with known issues | ||
|
|
||
| ### Migration Path |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can probably drop this section as the "migration path" isn't very long. Or just rewrite it to something like "tools that request this version of the JSON event stream schema will automatically have access to the new fields."
|
|
||
| ## Alternatives considered | ||
|
|
||
| ### Alternative Data Structures |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ### Alternative Data Structures |
| ### Unconditionally include optional field | ||
| - We selected conditional inclusion to keep JSON output clean and avoid null values, | ||
| improving the developer experience for tools consuming the data. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ### Unconditionally include optional field | |
| - We selected conditional inclusion to keep JSON output clean and avoid null values, | |
| improving the developer experience for tools consuming the data. | |
| - **Unconditionally include optional fields**: We selected conditional inclusion | |
| to keep JSON output clean and avoid null values, improving the developer | |
| experience for tools consuming the data. |
| and it was decided that a seperata proposal should be made on how to represent | ||
| the time units in the name/value. | ||
|
|
||
| ### Potential Extensions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ### Potential Extensions |
|
|
||
| This enhancement establishes future richer tooling experiences: | ||
|
|
||
| ### Alternative Field Naming |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ### Alternative Field Naming |
| This proposal maintains full backward compatibility through careful design: | ||
|
|
||
| - **ABI Version Protection**: New fields are conditionally included based on ABI | ||
| version checks, ensuring older tools continue to function without modification | ||
| - **Experimental Feature Migration**: The existing experimental `_tags` field is | ||
| replaced with the `tags` array. Since experimental features don't provide | ||
| stability guarantees, this replacement doesn't constitute a breaking change | ||
| - **Graceful Degradation**: Tools that don't expect the new fields will simply ignore | ||
| them, while updated tools can leverage the enhanced metadata | ||
|
|
||
| No existing functionality is affected, making this a purely additive enhancement. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| This proposal maintains full backward compatibility through careful design: | |
| - **ABI Version Protection**: New fields are conditionally included based on ABI | |
| version checks, ensuring older tools continue to function without modification | |
| - **Experimental Feature Migration**: The existing experimental `_tags` field is | |
| replaced with the `tags` array. Since experimental features don't provide | |
| stability guarantees, this replacement doesn't constitute a breaking change | |
| - **Graceful Degradation**: Tools that don't expect the new fields will simply ignore | |
| them, while updated tools can leverage the enhanced metadata | |
| No existing functionality is affected, making this a purely additive enhancement. | |
| This proposal is additive only. Tools using earlier versions of the JSON event stream schema are not affected. |
| + ["url": <string>,] ; the bug URL | ||
| + ["id": <string>,] ; the bug id | ||
| + ["title": <string>] ; the human readable bug title | ||
| +} ; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| +} ; | |
| +} |
|
|
||
| ### Sample JSON Output | ||
|
|
||
| Given the following Test Case |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Given the following Test Case | |
| Given the following test case: |
|
|
||
| ### Implementation Strategy | ||
|
|
||
| Fields are only included when the test actually has at least one matching trait applied, preserving | ||
| backwards compatibility with previous versions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| ### Implementation Strategy | |
| Fields are only included when the test actually has at least one matching trait applied, preserving | |
| backwards compatibility with previous versions. |
| This enhancement builds upon the existing test metadata infrastructure already used | ||
| internally by Swift Testing. The implementation reuses established data structures, | ||
| ensuring consistency and minimizing complexity. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| This enhancement builds upon the existing test metadata infrastructure already used | |
| internally by Swift Testing. The implementation reuses established data structures, | |
| ensuring consistency and minimizing complexity. | |
| We propose adding `"tags"`, `"bugs"`, and `"timeLimit"` fields to the existing `<test-suite>` and `<test-function>` structures in Swift Testing's JSON event stream schema starting with version `"6.3"`. |
|
|
||
| These additions leverage existing internal data structures, ensuring minimal performance | ||
| impact while maximizing the value delivered to external tools. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| These additions leverage existing internal data structures, ensuring minimal performance | |
| impact while maximizing the value delivered to external tools. |
|
|
||
| We propose enriching the test payload in the event JSON stream by adding three | ||
| metadata fields: | ||
|
|
||
| - **`tags`**: An array of strings where each item represents a single tag applied to the test, | ||
| enabling categorization and filtering | ||
| - **`bugs`**: An array of bug references, providing traceability between tests | ||
| and issue tracking | ||
| - **`timeLimit`**: The test's time limit in seconds, enabling performance monitoring | ||
| and timeout analysis |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| We propose enriching the test payload in the event JSON stream by adding three | |
| metadata fields: | |
| - **`tags`**: An array of strings where each item represents a single tag applied to the test, | |
| enabling categorization and filtering | |
| - **`bugs`**: An array of bug references, providing traceability between tests | |
| and issue tracking | |
| - **`timeLimit`**: The test's time limit in seconds, enabling performance monitoring | |
| and timeout analysis | |
| We propose adding new fields to Swift Testing's JSON event stream schema that represent the described traits. |
|
|
||
| By exposing this information, we unlock new possibilities for Swift Testing | ||
| tooling ecosystem. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| By exposing this information, we unlock new possibilities for Swift Testing | |
| tooling ecosystem. |
This is the problem statement, not the solution.
|
|
||
| Currently missing from the JSON output are: | ||
| - **Test tags**: Used for categorization | ||
| - **Bug associations**: Critical for tracking which tests verify specific bug fixes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - **Bug associations**: Critical for tracking which tests verify specific bug fixes | |
| - **Bug associations**: Track bugs associated with specific tests |
| Currently missing from the JSON output are: | ||
| - **Test tags**: Used for categorization | ||
| - **Bug associations**: Critical for tracking which tests verify specific bug fixes | ||
| - **Time limits**: Essential for performance monitoring and timeout management |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| - **Time limits**: Essential for performance monitoring and timeout management | |
| - **Time limits**: Useful for performance monitoring and timeout management |
|
|
||
| ## Motivation | ||
|
|
||
| Swift Testing's event JSON stream provides data for external tooling, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Swift Testing's event JSON stream provides data for external tooling, | |
| Swift Testing's JSON event stream provides data for external tooling, |
This proposal adds test metadata (
tags,bugsandtimeLimit) to Swift Testing's event JSON ABI, enabling richer external tooling capabilities.Links: Implementation | Forum Pitch